enhancement(tag_cardinality_limit): Add per-tag cache_size_per_key override in probabilistic mode by ArunPiduguDD · Pull Request #25650 · vectordotdev/vector

ArunPiduguDD · 2026-06-18T01:35:04Z

Summary

In probabilistic mode, the bloom filter cache size (cache_size_per_key) was a single global/per-metric setting inherited by every tag. However, when specifying a per-tag override, it doesn't really make sense to inherit this value, especially in cases where the per-tag override specifies a much higher limit than the global/per-metric limit.

This adds an optional cache_size_per_key field to per-tag limit_override entries. When set, it overrides the bloom filter size for that tag only. When omitted, it inherits from the enclosing per-metric or global config as before. Setting it in exact mode causes a validation error.

Example:

mode: probabilistic
cache_size_per_key: 5120
per_tag_limits:
  trace_id:
    mode: limit_override
    value_limit: 1000
    cache_size_per_key: 32768  # bigger filter for this tag only

Vector configuration

See example above. Works in both top-level per_tag_limits and inside per_metric_limits.

How did you test this PR?

Added unit tests for apply_cache_size_override covering probabilistic+Some, probabilistic+None, and exact+Some (no-op).
Added YAML deserialization tests for both per_metric_limits and global per_tag_limits scopes.
All existing 41 tag cardinality tests continue to pass.

Change Type

Is this a breaking change?

Yes
No

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the no-changelog label to this PR.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce0e240502

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

pront · 2026-06-26T21:36:18Z

codex generated

Mostly yes, but I’d ask for one small follow-up before merging:

Finding

[P2] cache_size_per_key validation only rejects inherited exact, not inherited excluded. In src/transforms/tag_cardinality_limit/config.rs, per-metric validation only checks OverrideMode::Exact; a per-metric config with mode: excluded plus a per-tag cache_size_per_key would still build, even though the changelog/docs now say the option is only valid in probabilistic mode. Tiny fix: reject any per-metric inherited mode that is not OverrideMode::Probabilistic(_), or explicitly document that excluded metrics ignore nested per-tag settings.

Minor polish:

The PR body still says exact mode is “silently ignored,” but the code/changelog now say it errors.
A couple comments/docs still imply all per-tag settings are inherited, e.g. src/transforms/tag_cardinality_limit/mod.rs. Worth rewording so cache_size_per_key is called out as the exception.

Verification: cargo test -p vector --lib --no-default-features --features transforms-tag_cardinality_limit transforms::tag_cardinality_limit:: passes: 45/45. git diff --check is clean. The earlier generated-docs churn also looks fixed now.

ArunPiduguDD · 2026-06-29T14:50:02Z

Updated PR with requested changes

pront · 2026-06-29T15:24:16Z

Updated PR with requested changes

@ArunPiduguDD thanks. Please also fix the merge conflicts and I will take a final look.

…in probabilistic mode

…_key in exact mode

…abilistic mode

ArunPiduguDD · 2026-06-29T15:38:11Z

Updated PR with requested changes

@ArunPiduguDD thanks. Please also fix the merge conflicts and I will take a final look.

@pront Rebased PR & fixed merged conflicts (+signed earlier unsigned commits)

ArunPiduguDD marked this pull request as ready for review June 18, 2026 02:36

ArunPiduguDD requested review from a team as code owners June 18, 2026 02:36

ArunPiduguDD force-pushed the feature/tag-cardinality-per-tag-cache-size branch from 3bc46fc to b845b1c Compare June 18, 2026 02:47

github-actions Bot added the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jun 18, 2026

ArunPiduguDD force-pushed the feature/tag-cardinality-per-tag-cache-size branch 2 times, most recently from 9850e50 to ce0e240 Compare June 18, 2026 02:49

chatgpt-codex-connector Bot reviewed Jun 18, 2026

View reviewed changes

Comment thread website/cue/reference/generated/configuration.cue

ArunPiduguDD force-pushed the feature/tag-cardinality-per-tag-cache-size branch 2 times, most recently from 98dc579 to 657934e Compare June 18, 2026 03:56

dd-sebastien-lb approved these changes Jun 18, 2026

View reviewed changes

ArunPiduguDD changed the title ~~feat(tag_cardinality_limit): add per-tag cache_size_per_key override in probabilistic mode~~ enhancement(tag_cardinality_limit): add per-tag cache_size_per_key override in probabilistic mode Jun 18, 2026

ArunPiduguDD changed the title ~~enhancement(tag_cardinality_limit): add per-tag cache_size_per_key override in probabilistic mode~~ enhancement(tag_cardinality_limit): Add per-tag cache_size_per_key override in probabilistic mode Jun 18, 2026

rtrieu self-requested a review June 22, 2026 18:22

rtrieu approved these changes Jun 22, 2026

View reviewed changes

pront reviewed Jun 23, 2026

View reviewed changes

Comment thread changelog.d/tag_cardinality_per_tag_cache_size.enhancement.md

ArunPiduguDD requested a review from pront June 24, 2026 17:35

ArunPiduguDD added 6 commits June 29, 2026 15:34

feat(tag_cardinality_limit): add per-tag cache_size_per_key override …

225e490

…in probabilistic mode

chore: regenerate component docs

7679bd7

feat(tag_cardinality_limit): add config validation for cache_size_per…

c56f39d

…_key in exact mode

changelog: update tag_cardinality_limit cache_size_per_key fragment

40b10c5

fix(tag_cardinality_limit): reject cache_size_per_key in any non-prob…

68fe2f2

…abilistic mode

Update docs/comments

2ab0c0f

ArunPiduguDD force-pushed the feature/tag-cardinality-per-tag-cache-size branch from 3536cf6 to 2ab0c0f Compare June 29, 2026 15:37

pront approved these changes Jun 29, 2026

View reviewed changes

github-actions Bot removed the docs review on hold The documentation team reviews PRs only after a PR is approved by the COSE team. label Jun 29, 2026

ArunPiduguDD added this pull request to the merge queue Jun 29, 2026

Merged via the queue into master with commit be2b50c Jun 29, 2026
84 checks passed

ArunPiduguDD deleted the feature/tag-cardinality-per-tag-cache-size branch June 29, 2026 17:44

github-actions Bot locked and limited conversation to collaborators Jun 29, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

enhancement(tag_cardinality_limit): Add per-tag cache_size_per_key override in probabilistic mode#25650

enhancement(tag_cardinality_limit): Add per-tag cache_size_per_key override in probabilistic mode#25650
ArunPiduguDD merged 6 commits into
masterfrom
feature/tag-cardinality-per-tag-cache-size

ArunPiduguDD commented Jun 18, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

Uh oh!

Uh oh!

pront commented Jun 26, 2026

Uh oh!

ArunPiduguDD commented Jun 29, 2026

Uh oh!

pront commented Jun 29, 2026

Uh oh!

ArunPiduguDD commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

ArunPiduguDD commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Vector configuration

How did you test this PR?

Change Type

Is this a breaking change?

Does this PR include user facing changes?

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

pront commented Jun 26, 2026

Uh oh!

ArunPiduguDD commented Jun 29, 2026

Uh oh!

pront commented Jun 29, 2026

Uh oh!

ArunPiduguDD commented Jun 29, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ArunPiduguDD commented Jun 18, 2026 •

edited

Loading